Maximizing CNN Validation Performance - Political Meme Classification

Introduction

This project focuses on designing and optimizing a convolutional neural network (CNN) for the classification of political memes into conservative and liberal viewpoints, aiming to achieve the highest possible accuracy on a validation set. The process includes employing techniques like early stopping and model checkpointing to monitor performance, and making adjustments to the CNN architecture based on changes in accuracy related to hyperparameters such as filters and layers. The task emphasizes reproducibility through a pre-determined data split and involves the creation of separate data generators for training, validation, and testing. The project’s success will be ultimately evaluated on an independent test set, with learning curves and detailed analysis included in the final report.

Dataset Source

For this project, the image dataset was sourced by exploring meme-focused pages on social platforms such as Reddit, Facebook, and Pinterest. A total of 1,000 images were collected, with an equal distribution between conservative and liberal political themes.

Special thanks to Kate Arendes for contributions to the collection process.

Loading Python Libraries

import PIL
import numpy as np
from PIL import Image
from keras import layers
from tensorflow import keras
from keras import regularizers
from google.colab import drive
import matplotlib.pyplot as plt
from keras.metrics import Precision
from keras.preprocessing.image import ImageDataGenerator
from sklearn.metrics import accuracy_score, precision_score

Load the data: The Political Memes Classification Dataset

• First, lets download the images from google drive:

# Let's mount the drive to load the images
drive.mount('/content/drive')

Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

Dataset Generation - Image Data Generator

# Let's set the base directory for loading the political meme images
base_directory = "/content/drive/My Drive/Political Meme Dataset/"

# Let's initialize the ImageDataGenerator with rescaling to normalize pixel values
my_generator = ImageDataGenerator(rescale=1./255)

# Let's set up the training data generator
# This loads images of size 150x150, in batches of 4, with binary class labels
train_generator = my_generator.flow_from_directory(
    f"{base_directory}/training/",
    target_size=(150, 150),
    batch_size=4,
    class_mode='binary'
)

# Let's set up the validation data generator
# Loads images of the same size and batch size as the training generator
valid_generator = my_generator.flow_from_directory(
    f"{base_directory}/validation/",
    target_size=(150, 150),
    batch_size=4,
    class_mode='binary'
)

# Let's set up the test data generator
# Uses the same parameters for consistency across training, validation, and testing
test_generator = my_generator.flow_from_directory(
    f"{base_directory}/test/",
    target_size=(150, 150),
    batch_size=4,
    class_mode='binary'
)

Found 600 images belonging to 2 classes.
Found 200 images belonging to 2 classes.
Found 200 images belonging to 2 classes.

Data Visualization

# Let's load a single image using PIL library.
image = Image.open(f"{base_directory}/training/train_liberal/0f76446d7d65a9e6508a226ae33e8a51--felder-donald-oconnor.jpg")

# Let's get some details about the image.
print("Image Mode   -->", image.mode)
print("Image Format --> ", image.format)
print("Image Size   -->", image.size)

Image Mode   --> RGB
Image Format -->  JPEG
Image Size   --> (118, 108)

# Let's display the colored image
plt.imshow(np.asarray(image))
plt.colorbar()

<matplotlib.colorbar.Colorbar at 0x7c726ab276a0>

# Let's convert the input image to grayscale
gs_image = image.convert(mode='L')

# Let's display the grayscale image using matplotlib
plt.imshow(np.asarray(gs_image), cmap='gray')

<matplotlib.image.AxesImage at 0x7c726c344cd0>

# Let's resize the image to 200x200 pixels
img_resized = image.resize((200,200))

# Let's print the size of the resized image to verify the new dimensions
print(img_resized.size)

# Let's display the resized image using matplotlib
plt.imshow(np.asarray(img_resized))

(200, 200)

<matplotlib.image.AxesImage at 0x7c726c1bdf00>

Train Images - Data Visualization

# Let's loop through batches of images from the train generator
for my_batch in train_generator:
    images = my_batch[0]
    labels = my_batch[1]

    # Let's iterate over each image and its corresponding label in the batch
    for i in range(len(labels)):
        plt.imshow(images[i])
        plt.colorbar()
        plt.show()

        # Let's print the label associated with the image
        print(labels[i])
    break

1.0

0.0

0.0

0.0

Validation Images - Data Visualization

# Let's loop through batches of images from the validation generator
for my_batch in valid_generator:
    images = my_batch[0]
    labels = my_batch[1]

    # Let's iterate over each image and its corresponding label in the batch
    for i in range(len(labels)):
        plt.imshow(images[i])
        plt.colorbar()
        plt.show()

        # Let's print the label associated with the image
        print(labels[i])
    break

0.0

1.0

1.0

0.0

Test Images - Data Visualization

# Let's loop through batches of images from the test generator
for my_batch in test_generator:
    images = my_batch[0]
    labels = my_batch[1]

    # Let's iterate over each image and its corresponding label in the batch
    for i in range(len(labels)):
        plt.imshow(images[i])
        plt.colorbar()
        plt.show()

        # Let's print the label associated with the image
        print(labels[i])
    break

0.0

0.0

0.0

0.0

Model Building

# Define the input shape and number of classes
input_shape = (150, 150, 3)
num_classes = 2

# Start defining the model
inputs = keras.Input(shape=input_shape)
x = layers.Conv2D(32, 3, padding='same', activation='relu')(inputs)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(64, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(128, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(128, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Adding a couple more Conv2D and MaxPooling2D layers
x = layers.Conv2D(256, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Global Average Pooling followed by the classifier
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.5)(x)  # Common dropout rate for regularization

# Output layer
outputs = layers.Dense(1, activation='sigmoid')(x)

# Finalize the model
model = keras.Model(inputs=inputs, outputs=outputs)
model.summary()

Model: "model_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_2 (InputLayer)        [(None, 150, 150, 3)]     0         
                                                                 
 conv2d_17 (Conv2D)          (None, 150, 150, 32)      896       
                                                                 
 max_pooling2d_17 (MaxPooli  (None, 75, 75, 32)        0         
 ng2D)                                                           
                                                                 
 conv2d_18 (Conv2D)          (None, 75, 75, 64)        18496     
                                                                 
 max_pooling2d_18 (MaxPooli  (None, 37, 37, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_19 (Conv2D)          (None, 37, 37, 128)       73856     
                                                                 
 max_pooling2d_19 (MaxPooli  (None, 18, 18, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_20 (Conv2D)          (None, 18, 18, 128)       147584    
                                                                 
 max_pooling2d_20 (MaxPooli  (None, 9, 9, 128)         0         
 ng2D)                                                           
                                                                 
 conv2d_21 (Conv2D)          (None, 9, 9, 256)         295168    
                                                                 
 max_pooling2d_21 (MaxPooli  (None, 4, 4, 256)         0         
 ng2D)                                                           
                                                                 
 global_average_pooling2d_1  (None, 256)               0         
  (GlobalAveragePooling2D)                                       
                                                                 
 dropout_5 (Dropout)         (None, 256)               0         
                                                                 
 dense_9 (Dense)             (None, 1)                 257       
                                                                 
=================================================================
Total params: 536257 (2.05 MB)
Trainable params: 536257 (2.05 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

# Let's compile the CNN model using binary cross_entropy as loss function and adam as optimizer
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy', keras.metrics.Precision()])

# Let's define the callbacks for Model saving and Early stopping

cb_check = keras.callbacks.ModelCheckpoint(
    filepath="checkpoint_filepath",
    save_best_only=True,
    monitor="val_loss")

cb_early = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=30
)

# Let's train and validate model on the training and validation data

history = model.fit(train_generator, validation_data = valid_generator, epochs = 10, verbose = 1, batch_size = 8, callbacks = [cb_check, cb_early])

Epoch 1/10
150/150 [==============================] - 7s 36ms/step - loss: 0.6890 - accuracy: 0.5767 - precision_1: 0.5821 - val_loss: 0.6557 - val_accuracy: 0.6200 - val_precision_1: 0.5682
Epoch 2/10
150/150 [==============================] - 5s 32ms/step - loss: 0.5482 - accuracy: 0.7567 - precision_1: 0.7188 - val_loss: 0.3334 - val_accuracy: 0.9400 - val_precision_1: 0.8929
Epoch 3/10
150/150 [==============================] - 5s 32ms/step - loss: 0.4263 - accuracy: 0.8567 - precision_1: 0.8242 - val_loss: 0.2132 - val_accuracy: 0.9450 - val_precision_1: 0.9159
Epoch 4/10
150/150 [==============================] - 5s 32ms/step - loss: 0.3202 - accuracy: 0.8750 - precision_1: 0.8462 - val_loss: 0.2123 - val_accuracy: 0.9050 - val_precision_1: 0.9551
Epoch 5/10
150/150 [==============================] - 5s 33ms/step - loss: 0.2947 - accuracy: 0.8983 - precision_1: 0.8842 - val_loss: 0.1506 - val_accuracy: 0.9550 - val_precision_1: 0.9417
Epoch 6/10
150/150 [==============================] - 4s 25ms/step - loss: 0.2917 - accuracy: 0.8917 - precision_1: 0.8660 - val_loss: 0.2518 - val_accuracy: 0.8800 - val_precision_1: 0.8065
Epoch 7/10
150/150 [==============================] - 4s 25ms/step - loss: 0.2616 - accuracy: 0.9067 - precision_1: 0.9013 - val_loss: 0.2803 - val_accuracy: 0.8700 - val_precision_1: 0.7937
Epoch 8/10
150/150 [==============================] - 4s 24ms/step - loss: 0.2586 - accuracy: 0.9033 - precision_1: 0.8854 - val_loss: 0.2528 - val_accuracy: 0.8850 - val_precision_1: 0.8130
Epoch 9/10
150/150 [==============================] - 4s 25ms/step - loss: 0.2721 - accuracy: 0.8867 - precision_1: 0.8718 - val_loss: 0.1694 - val_accuracy: 0.9350 - val_precision_1: 0.8850
Epoch 10/10
150/150 [==============================] - 4s 26ms/step - loss: 0.2068 - accuracy: 0.9167 - precision_1: 0.9058 - val_loss: 0.1603 - val_accuracy: 0.9400 - val_precision_1: 0.8929

Model Evaluation(Precison, Recall, F1-score)

train_accuracy = history.history["accuracy"]
train_loss = history.history["loss"]
train_precision = history.history["precision_1"]
val_accuracy = history.history["val_accuracy"]
val_loss = history.history["val_loss"]
val_precision = history.history["val_precision_1"]
epochs = range(1, len(train_accuracy) + 1)
plt.plot(epochs, train_accuracy, "bo", label="Training accuracy")
plt.title("Training Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, train_loss, "bo", label="Training loss")
plt.title("Training Loss")
plt.legend()
plt.show()
plt.plot(epochs, train_precision, "bo", label="Training precision")
plt.title("Training Precision")
plt.legend()
plt.show()
plt.plot(epochs, val_accuracy, "bo", label="Validation accuracy")
plt.title("Validation Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, val_loss, "bo", label="Validation loss")
plt.title("Validation Loss")
plt.legend()
plt.show()
plt.plot(epochs, val_precision, "bo", label="Validation precision")
plt.title("Validation Precision")
plt.legend()
plt.show()

Model Inference

# Let's loads the best-performing model and evaluate on the test data

model = keras.models.load_model("checkpoint_filepath")
model.evaluate(test_generator)

50/50 [==============================] - 73s 1s/step - loss: 0.2016 - accuracy: 0.9500 - precision_1: 0.9245

[0.20159845054149628, 0.949999988079071, 0.9245283007621765]

The initial model architecture resulted in a training accuracy of 0.9167 and a validation accuracy of 0.94 after being trained for 10 epochs. The next steps include increasing the number of epochs to 50 to observe how the training and validation accuracies change across different epochs. This extended training period will help determine if the model is benefiting from more training time or if it begins to overfit the training data. Observing the trend in validation accuracy will also indicate whether the model generalizes well to unseen data. Additional measures, such as implementing early stopping or adjusting the learning rate, may be considered based on the outcomes observed at different epochs.

Let's Increase the Epochs

# Define the input shape and number of classes
input_shape = (150, 150, 3)
num_classes = 2

# Start defining the model
inputs = keras.Input(shape=input_shape)
x = layers.Conv2D(32, 3, padding='same', activation='relu')(inputs)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(64, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(128, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(128, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Adding a couple more Conv2D and MaxPooling2D layers
x = layers.Conv2D(256, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Global Average Pooling followed by the classifier
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.5)(x)  # Common dropout rate for regularization

# Output layer
outputs = layers.Dense(1, activation='sigmoid')(x)

# Finalize the model
model_increase_epochs = keras.Model(inputs=inputs, outputs=outputs)

# Let's compile the CNN model using binary cross_entropy as loss function and adam as optimizer
model_increase_epochs.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy', keras.metrics.Precision()])

# Let's define the callbacks for Model saving and Early stopping

cb_check = keras.callbacks.ModelCheckpoint(
    filepath="base_model_checkpoint_filepath",
    save_best_only=True,
    monitor="val_loss")

cb_early = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=30
)

# Let's train and validate model on the training and validation data

history_increase_epochs = model_increase_epochs.fit(train_generator, validation_data = valid_generator, epochs = 30, verbose = 1, batch_size = 8, callbacks = [cb_check, cb_early])

Epoch 1/30
150/150 [==============================] - 7s 34ms/step - loss: 0.6025 - accuracy: 0.6800 - precision_2: 0.7673 - val_loss: 0.2778 - val_accuracy: 0.9400 - val_precision_2: 0.9314
Epoch 2/30
150/150 [==============================] - 4s 25ms/step - loss: 0.4617 - accuracy: 0.8133 - precision_2: 0.7749 - val_loss: 0.2888 - val_accuracy: 0.9550 - val_precision_2: 0.9333
Epoch 3/30
150/150 [==============================] - 5s 32ms/step - loss: 0.4052 - accuracy: 0.8600 - precision_2: 0.8396 - val_loss: 0.1892 - val_accuracy: 0.9550 - val_precision_2: 0.9417
Epoch 4/30
150/150 [==============================] - 5s 34ms/step - loss: 0.3427 - accuracy: 0.8683 - precision_2: 0.8421 - val_loss: 0.1826 - val_accuracy: 0.9400 - val_precision_2: 0.8929
Epoch 5/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2917 - accuracy: 0.8833 - precision_2: 0.8571 - val_loss: 0.3593 - val_accuracy: 0.8400 - val_precision_2: 0.9722
Epoch 6/30
150/150 [==============================] - 4s 26ms/step - loss: 0.3290 - accuracy: 0.8750 - precision_2: 0.8571 - val_loss: 0.2781 - val_accuracy: 0.8900 - val_precision_2: 0.8197
Epoch 7/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2959 - accuracy: 0.8983 - precision_2: 0.8794 - val_loss: 0.1890 - val_accuracy: 0.9350 - val_precision_2: 0.9394
Epoch 8/30
150/150 [==============================] - 5s 32ms/step - loss: 0.3395 - accuracy: 0.8550 - precision_2: 0.8257 - val_loss: 0.1690 - val_accuracy: 0.9700 - val_precision_2: 0.9519
Epoch 9/30
150/150 [==============================] - 5s 32ms/step - loss: 0.2545 - accuracy: 0.9167 - precision_2: 0.8956 - val_loss: 0.1290 - val_accuracy: 0.9550 - val_precision_2: 0.9174
Epoch 10/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2300 - accuracy: 0.9133 - precision_2: 0.9000 - val_loss: 0.1414 - val_accuracy: 0.9400 - val_precision_2: 0.8929
Epoch 11/30
150/150 [==============================] - 5s 33ms/step - loss: 0.2220 - accuracy: 0.9167 - precision_2: 0.9058 - val_loss: 0.1166 - val_accuracy: 0.9700 - val_precision_2: 0.9519
Epoch 12/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1804 - accuracy: 0.9317 - precision_2: 0.9191 - val_loss: 0.2790 - val_accuracy: 0.8800 - val_precision_2: 0.8065
Epoch 13/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1969 - accuracy: 0.9233 - precision_2: 0.9097 - val_loss: 0.1492 - val_accuracy: 0.9500 - val_precision_2: 0.9091
Epoch 14/30
150/150 [==============================] - 5s 33ms/step - loss: 0.1829 - accuracy: 0.9300 - precision_2: 0.9188 - val_loss: 0.1068 - val_accuracy: 0.9650 - val_precision_2: 0.9429
Epoch 15/30
150/150 [==============================] - 5s 34ms/step - loss: 0.1340 - accuracy: 0.9567 - precision_2: 0.9536 - val_loss: 0.1021 - val_accuracy: 0.9600 - val_precision_2: 0.9340
Epoch 16/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1566 - accuracy: 0.9467 - precision_2: 0.9408 - val_loss: 0.3301 - val_accuracy: 0.8950 - val_precision_2: 0.8264
Epoch 17/30
150/150 [==============================] - 4s 27ms/step - loss: 0.1869 - accuracy: 0.9333 - precision_2: 0.9248 - val_loss: 0.1835 - val_accuracy: 0.9350 - val_precision_2: 0.8850
Epoch 18/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1433 - accuracy: 0.9483 - precision_2: 0.9529 - val_loss: 0.1376 - val_accuracy: 0.9400 - val_precision_2: 0.8929
Epoch 19/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1453 - accuracy: 0.9467 - precision_2: 0.9379 - val_loss: 0.1894 - val_accuracy: 0.9350 - val_precision_2: 0.8850
Epoch 20/30
150/150 [==============================] - 5s 36ms/step - loss: 0.1103 - accuracy: 0.9517 - precision_2: 0.9593 - val_loss: 0.0948 - val_accuracy: 0.9750 - val_precision_2: 0.9798
Epoch 21/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0834 - accuracy: 0.9667 - precision_2: 0.9667 - val_loss: 0.2334 - val_accuracy: 0.9250 - val_precision_2: 0.8696
Epoch 22/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0864 - accuracy: 0.9650 - precision_2: 0.9604 - val_loss: 0.1216 - val_accuracy: 0.9500 - val_precision_2: 0.9245
Epoch 23/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0735 - accuracy: 0.9750 - precision_2: 0.9703 - val_loss: 0.1159 - val_accuracy: 0.9650 - val_precision_2: 0.9346
Epoch 24/30
150/150 [==============================] - 5s 33ms/step - loss: 0.0852 - accuracy: 0.9617 - precision_2: 0.9632 - val_loss: 0.0707 - val_accuracy: 0.9700 - val_precision_2: 0.9434
Epoch 25/30
150/150 [==============================] - 5s 33ms/step - loss: 0.0523 - accuracy: 0.9833 - precision_2: 0.9801 - val_loss: 0.0654 - val_accuracy: 0.9800 - val_precision_2: 0.9800
Epoch 26/30
150/150 [==============================] - 4s 27ms/step - loss: 0.0563 - accuracy: 0.9783 - precision_2: 0.9767 - val_loss: 0.1835 - val_accuracy: 0.9500 - val_precision_2: 0.9091
Epoch 27/30
150/150 [==============================] - 4s 24ms/step - loss: 0.0925 - accuracy: 0.9617 - precision_2: 0.9601 - val_loss: 0.1404 - val_accuracy: 0.9500 - val_precision_2: 0.9167
Epoch 28/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0781 - accuracy: 0.9733 - precision_2: 0.9702 - val_loss: 0.0910 - val_accuracy: 0.9700 - val_precision_2: 0.9434
Epoch 29/30
150/150 [==============================] - 4s 27ms/step - loss: 0.0469 - accuracy: 0.9817 - precision_2: 0.9801 - val_loss: 0.1405 - val_accuracy: 0.9550 - val_precision_2: 0.9252
Epoch 30/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0180 - accuracy: 0.9983 - precision_2: 0.9967 - val_loss: 0.1724 - val_accuracy: 0.9650 - val_precision_2: 0.9429

Model Evaluation(Precison, Recall, F1-score) - 30 epochs

train_accuracy = history_increase_epochs.history["accuracy"]
train_loss = history_increase_epochs.history["loss"]
train_precision = history_increase_epochs.history["precision_2"]
val_accuracy = history_increase_epochs.history["val_accuracy"]
val_loss = history_increase_epochs.history["val_loss"]
val_precision = history_increase_epochs.history["val_precision_2"]
epochs = range(1, len(train_accuracy) + 1)
plt.plot(epochs, train_accuracy, "bo", label="Training accuracy")
plt.title("Training Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, train_loss, "bo", label="Training loss")
plt.title("Training Loss")
plt.legend()
plt.show()
plt.plot(epochs, train_precision, "bo", label="Training precision")
plt.title("Training Precision")
plt.legend()
plt.show()
plt.plot(epochs, val_accuracy, "bo", label="Validation accuracy")
plt.title("Validation Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, val_loss, "bo", label="Validation loss")
plt.title("Validation Loss")
plt.legend()
plt.show()
plt.plot(epochs, val_precision, "bo", label="Validation precision")
plt.title("Validation Precision")
plt.legend()
plt.show()

Model Inference

# Let's loads the best-performing model and evaluate on the test data

best_model = keras.models.load_model("base_model_checkpoint_filepath")
best_model.evaluate(test_generator)

50/50 [==============================] - 1s 16ms/step - loss: 0.1484 - accuracy: 0.9550 - precision_2: 0.9333

[0.14840058982372284, 0.9549999833106995, 0.9333333373069763]

Increasing the number of epochs from 10 to 30 resulted in an increase in training accuracy from 0.9169 to 0.9983. Similarly, the validation accuracy increased from 0.94 to 0.97. As expected, the time required for training increased significantly from 3 minutes to 5 minutes due to the higher number of epochs. This indicates that the model benefited from additional training, as evidenced by the improvements in both training and validation accuracies.

Decrease the number of layers

# Define the input shape and number of classes
input_shape = (150, 150, 3)
num_classes = 2

# Start defining the model
inputs = keras.Input(shape=input_shape)
x = layers.Conv2D(32, 3, padding='same', activation='relu')(inputs)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(64, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(128, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Adding a couple more Conv2D and MaxPooling2D layers
x = layers.Conv2D(256, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Global Average Pooling followed by the classifier
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.5)(x)  # Common dropout rate for regularization

# Output layer
outputs = layers.Dense(1, activation='sigmoid')(x)

# Finalize the model
model_decrease_layers = keras.Model(inputs=inputs, outputs=outputs)

# Let's compile the CNN model using binary cross_entropy as loss function and adam as optimizer
model_decrease_layers.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy', keras.metrics.Precision()])

# Let's define the callbacks for Model saving and Early stopping

cb_check = keras.callbacks.ModelCheckpoint(
    filepath="decrease_layers_checkpoint_filepath",
    save_best_only=True,
    monitor="val_loss")

cb_early = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=30
)

# Let's train and validate model on the training and validation data

history_decrease_layers = model_decrease_layers.fit(train_generator, validation_data = valid_generator, epochs = 30, verbose = 1, batch_size = 8, callbacks = [cb_check, cb_early])

Epoch 1/30
150/150 [==============================] - 7s 36ms/step - loss: 0.6384 - accuracy: 0.6117 - precision_6: 0.6683 - val_loss: 0.3889 - val_accuracy: 0.8500 - val_precision_6: 0.9268
Epoch 2/30
150/150 [==============================] - 5s 31ms/step - loss: 0.5383 - accuracy: 0.8133 - precision_6: 0.7848 - val_loss: 0.2210 - val_accuracy: 0.9300 - val_precision_6: 0.9216
Epoch 3/30
150/150 [==============================] - 4s 25ms/step - loss: 0.4117 - accuracy: 0.8483 - precision_6: 0.8047 - val_loss: 0.3755 - val_accuracy: 0.8250 - val_precision_6: 0.7407
Epoch 4/30
150/150 [==============================] - 5s 31ms/step - loss: 0.3692 - accuracy: 0.8600 - precision_6: 0.8214 - val_loss: 0.1845 - val_accuracy: 0.9300 - val_precision_6: 0.9574
Epoch 5/30
150/150 [==============================] - 4s 26ms/step - loss: 0.3661 - accuracy: 0.8850 - precision_6: 0.8468 - val_loss: 0.1889 - val_accuracy: 0.9600 - val_precision_6: 0.9423
Epoch 6/30
150/150 [==============================] - 4s 25ms/step - loss: 0.3195 - accuracy: 0.8883 - precision_6: 0.8585 - val_loss: 0.2010 - val_accuracy: 0.9350 - val_precision_6: 0.8850
Epoch 7/30
150/150 [==============================] - 5s 31ms/step - loss: 0.3544 - accuracy: 0.8900 - precision_6: 0.8503 - val_loss: 0.1639 - val_accuracy: 0.9700 - val_precision_6: 0.9608
Epoch 8/30
150/150 [==============================] - 4s 26ms/step - loss: 0.3111 - accuracy: 0.8867 - precision_6: 0.8558 - val_loss: 0.3436 - val_accuracy: 0.8400 - val_precision_6: 0.7576
Epoch 9/30
150/150 [==============================] - 5s 31ms/step - loss: 0.2674 - accuracy: 0.9050 - precision_6: 0.8932 - val_loss: 0.1489 - val_accuracy: 0.9450 - val_precision_6: 0.9495
Epoch 10/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2787 - accuracy: 0.8900 - precision_6: 0.8634 - val_loss: 0.1836 - val_accuracy: 0.9600 - val_precision_6: 0.9340
Epoch 11/30
150/150 [==============================] - 5s 31ms/step - loss: 0.2507 - accuracy: 0.9100 - precision_6: 0.8968 - val_loss: 0.1213 - val_accuracy: 0.9700 - val_precision_6: 0.9519
Epoch 12/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2745 - accuracy: 0.8967 - precision_6: 0.8889 - val_loss: 0.2140 - val_accuracy: 0.9050 - val_precision_6: 0.8403
Epoch 13/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2572 - accuracy: 0.9083 - precision_6: 0.8889 - val_loss: 0.1525 - val_accuracy: 0.9600 - val_precision_6: 0.9259
Epoch 14/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2253 - accuracy: 0.9183 - precision_6: 0.9088 - val_loss: 0.1747 - val_accuracy: 0.9350 - val_precision_6: 0.8850
Epoch 15/30
150/150 [==============================] - 5s 31ms/step - loss: 0.2156 - accuracy: 0.9133 - precision_6: 0.8949 - val_loss: 0.1048 - val_accuracy: 0.9650 - val_precision_6: 0.9515
Epoch 16/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2030 - accuracy: 0.9300 - precision_6: 0.9135 - val_loss: 0.1429 - val_accuracy: 0.9400 - val_precision_6: 0.8929
Epoch 17/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1842 - accuracy: 0.9283 - precision_6: 0.9186 - val_loss: 0.1763 - val_accuracy: 0.9100 - val_precision_6: 0.8475
Epoch 18/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2090 - accuracy: 0.9183 - precision_6: 0.8959 - val_loss: 0.2044 - val_accuracy: 0.9000 - val_precision_6: 0.8333
Epoch 19/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1977 - accuracy: 0.9233 - precision_6: 0.9097 - val_loss: 0.1396 - val_accuracy: 0.9550 - val_precision_6: 0.9174
Epoch 20/30
150/150 [==============================] - 4s 26ms/step - loss: 0.2096 - accuracy: 0.9217 - precision_6: 0.9148 - val_loss: 0.1372 - val_accuracy: 0.9700 - val_precision_6: 0.9434
Epoch 21/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1763 - accuracy: 0.9383 - precision_6: 0.9283 - val_loss: 0.1483 - val_accuracy: 0.9450 - val_precision_6: 0.9009
Epoch 22/30
150/150 [==============================] - 5s 31ms/step - loss: 0.1603 - accuracy: 0.9383 - precision_6: 0.9311 - val_loss: 0.0925 - val_accuracy: 0.9700 - val_precision_6: 0.9434
Epoch 23/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1325 - accuracy: 0.9533 - precision_6: 0.9444 - val_loss: 0.1019 - val_accuracy: 0.9650 - val_precision_6: 0.9346
Epoch 24/30
150/150 [==============================] - 5s 34ms/step - loss: 0.1211 - accuracy: 0.9533 - precision_6: 0.9474 - val_loss: 0.0873 - val_accuracy: 0.9700 - val_precision_6: 0.9608
Epoch 25/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1849 - accuracy: 0.9333 - precision_6: 0.9167 - val_loss: 0.1551 - val_accuracy: 0.9300 - val_precision_6: 0.8772
Epoch 26/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1315 - accuracy: 0.9467 - precision_6: 0.9408 - val_loss: 0.1343 - val_accuracy: 0.9450 - val_precision_6: 0.9238
Epoch 27/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1170 - accuracy: 0.9617 - precision_6: 0.9601 - val_loss: 0.1024 - val_accuracy: 0.9650 - val_precision_6: 0.9346
Epoch 28/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0982 - accuracy: 0.9650 - precision_6: 0.9574 - val_loss: 0.2915 - val_accuracy: 0.8850 - val_precision_6: 0.8130
Epoch 29/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0881 - accuracy: 0.9667 - precision_6: 0.9605 - val_loss: 0.1394 - val_accuracy: 0.9400 - val_precision_6: 0.8929
Epoch 30/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0770 - accuracy: 0.9683 - precision_6: 0.9637 - val_loss: 0.1180 - val_accuracy: 0.9500 - val_precision_6: 0.9500

Model Evaluation(Precison, Recall, F1-score) - Decrease convolution layers

train_accuracy = history_decrease_layers.history["accuracy"]
train_loss = history_decrease_layers.history["loss"]
train_precision = history_decrease_layers.history["precision_6"]
val_accuracy = history_decrease_layers.history["val_accuracy"]
val_loss = history_decrease_layers.history["val_loss"]
val_precision = history_decrease_layers.history["val_precision_6"]
epochs = range(1, len(train_accuracy) + 1)
plt.plot(epochs, train_accuracy, "bo", label="Training accuracy")
plt.title("Training Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, train_loss, "bo", label="Training loss")
plt.title("Training Loss")
plt.legend()
plt.show()
plt.plot(epochs, train_precision, "bo", label="Training precision")
plt.title("Training Precision")
plt.legend()
plt.show()
plt.plot(epochs, val_accuracy, "bo", label="Validation accuracy")
plt.title("Validation Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, val_loss, "bo", label="Validation loss")
plt.title("Validation Loss")
plt.legend()
plt.show()
plt.plot(epochs, val_precision, "bo", label="Validation precision")
plt.title("Validation Precision")
plt.legend()
plt.show()

Model Inference

# Let's loads the best-performing model and evaluate on the test data

model = keras.models.load_model("decrease_layers_checkpoint_filepath")
model.evaluate(test_generator)

50/50 [==============================] - 1s 15ms/step - loss: 0.1241 - accuracy: 0.9400 - precision_6: 0.9000

[0.12408556789159775, 0.9399999976158142, 0.8999999761581421]

Surprisingly, decreasing one convolution and pooling layers by resulted in a training accuracy of 0.9683 and a validation accuracy of 0.95. The time it took to complete training and validation for 30 epochs was almost 3 minutes. The test performance is similar to the configuration with more layers; the accuracy with fewer layers is 0.94, while the accuracy with more layers is 0.9550.

Increase the number of convolutional filters

# Define the input shape and number of classes
input_shape = (150, 150, 3)
num_classes = 2

# Start defining the model
inputs = keras.Input(shape=input_shape)
x = layers.Conv2D(32, 3, padding='same', activation='relu')(inputs)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(64, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(256, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(256, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Adding a couple more Conv2D and MaxPooling2D layers
x = layers.Conv2D(256, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Global Average Pooling followed by the classifier
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.5)(x)  # Common dropout rate for regularization

# Output layer
outputs = layers.Dense(1, activation='sigmoid')(x)

# Finalize the model
model_increased_filters = keras.Model(inputs=inputs, outputs=outputs)

# Let's compile the CNN model using binary cross_entropy as loss function and adam as optimizer
model_increased_filters.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy', keras.metrics.Precision()])

# Let's define the callbacks for Model saving and Early stopping

cb_check = keras.callbacks.ModelCheckpoint(
    filepath="increase_filters_checkpoint_filepath",
    save_best_only=True,
    monitor="val_loss")

cb_early = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=30
)

# Let's train and validate model on the training and validation data

history_increased_filters = model_increased_filters.fit(train_generator, validation_data = valid_generator, epochs = 30, verbose = 1, batch_size = 8, callbacks = [cb_check, cb_early])

Epoch 1/30
150/150 [==============================] - 7s 35ms/step - loss: 0.6311 - accuracy: 0.6733 - precision_9: 0.6711 - val_loss: 0.3597 - val_accuracy: 0.9000 - val_precision_9: 0.8448
Epoch 2/30
150/150 [==============================] - 4s 25ms/step - loss: 0.5611 - accuracy: 0.7550 - precision_9: 0.7029 - val_loss: 0.6817 - val_accuracy: 0.5150 - val_precision_9: 0.5076
Epoch 3/30
150/150 [==============================] - 4s 26ms/step - loss: 0.6183 - accuracy: 0.7517 - precision_9: 0.6921 - val_loss: 0.4809 - val_accuracy: 0.8200 - val_precision_9: 0.7353
Epoch 4/30
150/150 [==============================] - 5s 35ms/step - loss: 0.5312 - accuracy: 0.7850 - precision_9: 0.7221 - val_loss: 0.2999 - val_accuracy: 0.9100 - val_precision_9: 0.9457
Epoch 5/30
150/150 [==============================] - 5s 33ms/step - loss: 0.4459 - accuracy: 0.8367 - precision_9: 0.7953 - val_loss: 0.2461 - val_accuracy: 0.9050 - val_precision_9: 0.8403
Epoch 6/30
150/150 [==============================] - 5s 33ms/step - loss: 0.3995 - accuracy: 0.8500 - precision_9: 0.8125 - val_loss: 0.1975 - val_accuracy: 0.9450 - val_precision_9: 0.9406
Epoch 7/30
150/150 [==============================] - 5s 33ms/step - loss: 0.3167 - accuracy: 0.8800 - precision_9: 0.8608 - val_loss: 0.1454 - val_accuracy: 0.9600 - val_precision_9: 0.9259
Epoch 8/30
150/150 [==============================] - 5s 34ms/step - loss: 0.2897 - accuracy: 0.8883 - precision_9: 0.8698 - val_loss: 0.1401 - val_accuracy: 0.9550 - val_precision_9: 0.9174
Epoch 9/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2840 - accuracy: 0.9033 - precision_9: 0.8878 - val_loss: 0.2197 - val_accuracy: 0.9100 - val_precision_9: 0.8475
Epoch 10/30
150/150 [==============================] - 4s 26ms/step - loss: 0.3070 - accuracy: 0.8967 - precision_9: 0.8719 - val_loss: 0.1498 - val_accuracy: 0.9600 - val_precision_9: 0.9340
Epoch 11/30
150/150 [==============================] - 5s 33ms/step - loss: 0.2452 - accuracy: 0.9083 - precision_9: 0.8990 - val_loss: 0.0774 - val_accuracy: 0.9850 - val_precision_9: 0.9709
Epoch 12/30
150/150 [==============================] - 4s 26ms/step - loss: 0.2598 - accuracy: 0.9200 - precision_9: 0.9145 - val_loss: 0.1180 - val_accuracy: 0.9800 - val_precision_9: 0.9615
Epoch 13/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2308 - accuracy: 0.9200 - precision_9: 0.9013 - val_loss: 0.1339 - val_accuracy: 0.9550 - val_precision_9: 0.9174
Epoch 14/30
150/150 [==============================] - 4s 26ms/step - loss: 0.2041 - accuracy: 0.9233 - precision_9: 0.9123 - val_loss: 0.1062 - val_accuracy: 0.9700 - val_precision_9: 0.9434
Epoch 15/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1899 - accuracy: 0.9300 - precision_9: 0.9216 - val_loss: 0.0901 - val_accuracy: 0.9750 - val_precision_9: 0.9524
Epoch 16/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1939 - accuracy: 0.9317 - precision_9: 0.9274 - val_loss: 0.0812 - val_accuracy: 0.9800 - val_precision_9: 0.9706
Epoch 17/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1993 - accuracy: 0.9317 - precision_9: 0.9302 - val_loss: 0.0855 - val_accuracy: 0.9700 - val_precision_9: 0.9434
Epoch 18/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1534 - accuracy: 0.9467 - precision_9: 0.9379 - val_loss: 0.1325 - val_accuracy: 0.9600 - val_precision_9: 0.9259
Epoch 19/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1672 - accuracy: 0.9383 - precision_9: 0.9340 - val_loss: 0.2987 - val_accuracy: 0.9100 - val_precision_9: 0.8475
Epoch 20/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1494 - accuracy: 0.9433 - precision_9: 0.9346 - val_loss: 0.1611 - val_accuracy: 0.9350 - val_precision_9: 0.8919
Epoch 21/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1508 - accuracy: 0.9517 - precision_9: 0.9502 - val_loss: 0.2378 - val_accuracy: 0.8950 - val_precision_9: 0.8264
Epoch 22/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1223 - accuracy: 0.9500 - precision_9: 0.9470 - val_loss: 0.1070 - val_accuracy: 0.9600 - val_precision_9: 0.9340
Epoch 23/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1350 - accuracy: 0.9533 - precision_9: 0.9474 - val_loss: 0.1106 - val_accuracy: 0.9600 - val_precision_9: 0.9259
Epoch 24/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1030 - accuracy: 0.9650 - precision_9: 0.9604 - val_loss: 0.0970 - val_accuracy: 0.9750 - val_precision_9: 0.9524
Epoch 25/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0725 - accuracy: 0.9767 - precision_9: 0.9735 - val_loss: 0.2406 - val_accuracy: 0.9100 - val_precision_9: 0.8475
Epoch 26/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1095 - accuracy: 0.9667 - precision_9: 0.9667 - val_loss: 0.2239 - val_accuracy: 0.8900 - val_precision_9: 0.8197
Epoch 27/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0797 - accuracy: 0.9767 - precision_9: 0.9735 - val_loss: 0.1151 - val_accuracy: 0.9400 - val_precision_9: 0.8929
Epoch 28/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0820 - accuracy: 0.9750 - precision_9: 0.9642 - val_loss: 0.1602 - val_accuracy: 0.9500 - val_precision_9: 0.9091
Epoch 29/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1663 - accuracy: 0.9467 - precision_9: 0.9379 - val_loss: 0.0975 - val_accuracy: 0.9650 - val_precision_9: 0.9346
Epoch 30/30
150/150 [==============================] - 5s 33ms/step - loss: 0.0960 - accuracy: 0.9667 - precision_9: 0.9575 - val_loss: 0.0504 - val_accuracy: 0.9750 - val_precision_9: 0.9524

Model Evaluation(Precison, Recall, F1-score) - Decrease convolution filters

train_accuracy = history_increased_filters.history["accuracy"]
train_loss = history_increased_filters.history["loss"]
train_precision = history_increased_filters.history["precision_9"]
val_accuracy = history_increased_filters.history["val_accuracy"]
val_loss = history_increased_filters.history["val_loss"]
val_precision = history_increased_filters.history["val_precision_9"]
epochs = range(1, len(train_accuracy) + 1)
plt.plot(epochs, train_accuracy, "bo", label="Training accuracy")
plt.title("Training Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, train_loss, "bo", label="Training loss")
plt.title("Training Loss")
plt.legend()
plt.show()
plt.plot(epochs, train_precision, "bo", label="Training precision")
plt.title("Training Precision")
plt.legend()
plt.show()
plt.plot(epochs, val_accuracy, "bo", label="Validation accuracy")
plt.title("Validation Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, val_loss, "bo", label="Validation loss")
plt.title("Validation Loss")
plt.legend()
plt.show()
plt.plot(epochs, val_precision, "bo", label="Validation precision")
plt.title("Validation Precision")
plt.legend()
plt.show()

Model Inference

# Let's loads the best-performing model and evaluate on the test data

model = keras.models.load_model("increase_filters_checkpoint_filepath")
model.evaluate(test_generator)

50/50 [==============================] - 1s 17ms/step - loss: 0.1671 - accuracy: 0.9450 - precision_9: 0.9009

[0.1671462506055832, 0.9449999928474426, 0.9009009003639221]

Surprisingly, increasing the number of convolution filters from 128 to 256 in two convolution layers resulted in a training accuracy of 0.9667 and a validation accuracy of 0.9750. The time it took to complete training and validation for 30 epochs was close to 3 minutes. The test performance is same to the configuration with more filters; the accuracy with fewer filters is 0.9450, while the accuracy with base configuration is 0.9550.

Decrease the image size

# Define the input shape and number of classes
input_shape = (100, 100, 3)
num_classes = 2

# Start defining the model
inputs = keras.Input(shape=input_shape)
x = layers.Conv2D(32, 3, padding='same', activation='relu')(inputs)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(64, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(256, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(256, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Adding a couple more Conv2D and MaxPooling2D layers
x = layers.Conv2D(256, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Global Average Pooling followed by the classifier
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.5)(x)  # Common dropout rate for regularization

# Output layer
outputs = layers.Dense(1, activation='sigmoid')(x)

# Finalize the model
model_decreased_image_size = keras.Model(inputs=inputs, outputs=outputs)

# Let's compile the CNN model using binary cross_entropy as loss function and adam as optimizer
model_decreased_image_size.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy', keras.metrics.Precision()])

# Let's define the callbacks for Model saving and Early stopping

cb_check = keras.callbacks.ModelCheckpoint(
    filepath="decrease_image_checkpoint_filepath",
    save_best_only=True,
    monitor="val_loss")

cb_early = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=30
)

# Let's train and validate model on the training and validation data

history_decreased_image_size = model_decreased_image_size.fit(train_generator, validation_data = valid_generator, epochs = 30, verbose = 1, batch_size = 8, callbacks = [cb_check, cb_early])

Epoch 1/30
150/150 [==============================] - 7s 35ms/step - loss: 0.6771 - accuracy: 0.5433 - precision_10: 0.5684 - val_loss: 0.4984 - val_accuracy: 0.8500 - val_precision_10: 0.9487
Epoch 2/30
150/150 [==============================] - 4s 25ms/step - loss: 0.6645 - accuracy: 0.6917 - precision_10: 0.6448 - val_loss: 0.6203 - val_accuracy: 0.8000 - val_precision_10: 0.8947
Epoch 3/30
150/150 [==============================] - 5s 33ms/step - loss: 0.5489 - accuracy: 0.7933 - precision_10: 0.7588 - val_loss: 0.2703 - val_accuracy: 0.9100 - val_precision_10: 0.8475
Epoch 4/30
150/150 [==============================] - 4s 25ms/step - loss: 0.4003 - accuracy: 0.8517 - precision_10: 0.8187 - val_loss: 0.3547 - val_accuracy: 0.8450 - val_precision_10: 0.7674
Epoch 5/30
150/150 [==============================] - 5s 33ms/step - loss: 0.3474 - accuracy: 0.8750 - precision_10: 0.8483 - val_loss: 0.2459 - val_accuracy: 0.9100 - val_precision_10: 0.8534
Epoch 6/30
150/150 [==============================] - 5s 35ms/step - loss: 0.3309 - accuracy: 0.8717 - precision_10: 0.8431 - val_loss: 0.1803 - val_accuracy: 0.9400 - val_precision_10: 0.9400
Epoch 7/30
150/150 [==============================] - 5s 33ms/step - loss: 0.2782 - accuracy: 0.8967 - precision_10: 0.8719 - val_loss: 0.1684 - val_accuracy: 0.9350 - val_precision_10: 0.9780
Epoch 8/30
150/150 [==============================] - 4s 26ms/step - loss: 0.3530 - accuracy: 0.8683 - precision_10: 0.8421 - val_loss: 0.1783 - val_accuracy: 0.9550 - val_precision_10: 0.9174
Epoch 9/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2585 - accuracy: 0.9150 - precision_10: 0.9055 - val_loss: 0.1725 - val_accuracy: 0.9350 - val_precision_10: 0.8850
Epoch 10/30
150/150 [==============================] - 5s 33ms/step - loss: 0.2449 - accuracy: 0.9083 - precision_10: 0.8964 - val_loss: 0.1183 - val_accuracy: 0.9750 - val_precision_10: 0.9524
Epoch 11/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2045 - accuracy: 0.9217 - precision_10: 0.9175 - val_loss: 0.2540 - val_accuracy: 0.8950 - val_precision_10: 0.8264
Epoch 12/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2040 - accuracy: 0.9250 - precision_10: 0.9264 - val_loss: 0.2036 - val_accuracy: 0.9100 - val_precision_10: 0.8475
Epoch 13/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2061 - accuracy: 0.9217 - precision_10: 0.9121 - val_loss: 0.1259 - val_accuracy: 0.9550 - val_precision_10: 0.9174
Epoch 14/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1878 - accuracy: 0.9333 - precision_10: 0.9392 - val_loss: 0.1239 - val_accuracy: 0.9650 - val_precision_10: 0.9346
Epoch 15/30
150/150 [==============================] - 5s 32ms/step - loss: 0.1955 - accuracy: 0.9250 - precision_10: 0.9293 - val_loss: 0.0922 - val_accuracy: 0.9700 - val_precision_10: 0.9434
Epoch 16/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1650 - accuracy: 0.9450 - precision_10: 0.9406 - val_loss: 0.1260 - val_accuracy: 0.9650 - val_precision_10: 0.9515
Epoch 17/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1498 - accuracy: 0.9467 - precision_10: 0.9467 - val_loss: 0.1421 - val_accuracy: 0.9750 - val_precision_10: 0.9612
Epoch 18/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1711 - accuracy: 0.9400 - precision_10: 0.9314 - val_loss: 0.1141 - val_accuracy: 0.9600 - val_precision_10: 0.9259
Epoch 19/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1386 - accuracy: 0.9533 - precision_10: 0.9533 - val_loss: 0.1124 - val_accuracy: 0.9700 - val_precision_10: 0.9519
Epoch 20/30
150/150 [==============================] - 5s 33ms/step - loss: 0.1379 - accuracy: 0.9483 - precision_10: 0.9410 - val_loss: 0.0620 - val_accuracy: 0.9750 - val_precision_10: 0.9524
Epoch 21/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1545 - accuracy: 0.9433 - precision_10: 0.9375 - val_loss: 0.2216 - val_accuracy: 0.9100 - val_precision_10: 0.8475
Epoch 22/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1055 - accuracy: 0.9550 - precision_10: 0.9565 - val_loss: 0.0851 - val_accuracy: 0.9650 - val_precision_10: 0.9429
Epoch 23/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1115 - accuracy: 0.9600 - precision_10: 0.9600 - val_loss: 0.0859 - val_accuracy: 0.9650 - val_precision_10: 0.9346
Epoch 24/30
150/150 [==============================] - 4s 24ms/step - loss: 0.0806 - accuracy: 0.9733 - precision_10: 0.9671 - val_loss: 0.1564 - val_accuracy: 0.9600 - val_precision_10: 0.9259
Epoch 25/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0836 - accuracy: 0.9700 - precision_10: 0.9669 - val_loss: 0.1628 - val_accuracy: 0.9550 - val_precision_10: 0.9174
Epoch 26/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0939 - accuracy: 0.9700 - precision_10: 0.9638 - val_loss: 0.0989 - val_accuracy: 0.9750 - val_precision_10: 0.9524
Epoch 27/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0828 - accuracy: 0.9700 - precision_10: 0.9608 - val_loss: 0.1323 - val_accuracy: 0.9600 - val_precision_10: 0.9259
Epoch 28/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1188 - accuracy: 0.9583 - precision_10: 0.9421 - val_loss: 0.1243 - val_accuracy: 0.9400 - val_precision_10: 0.8929
Epoch 29/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1644 - accuracy: 0.9500 - precision_10: 0.9441 - val_loss: 0.0734 - val_accuracy: 0.9800 - val_precision_10: 0.9615
Epoch 30/30
150/150 [==============================] - 5s 33ms/step - loss: 0.1053 - accuracy: 0.9650 - precision_10: 0.9666 - val_loss: 0.0532 - val_accuracy: 0.9900 - val_precision_10: 0.9804

Model Evaluation(Precison, Recall, F1-score) - Decreased Image Size

train_accuracy = history_decreased_image_size.history["accuracy"]
train_loss = history_decreased_image_size.history["loss"]
train_precision = history_decreased_image_size.history["precision_10"]
val_accuracy = history_decreased_image_size.history["val_accuracy"]
val_loss = history_decreased_image_size.history["val_loss"]
val_precision = history_decreased_image_size.history["val_precision_10"]
epochs = range(1, len(train_accuracy) + 1)
plt.plot(epochs, train_accuracy, "bo", label="Training accuracy")
plt.title("Training Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, train_loss, "bo", label="Training loss")
plt.title("Training Loss")
plt.legend()
plt.show()
plt.plot(epochs, train_precision, "bo", label="Training precision")
plt.title("Training Precision")
plt.legend()
plt.show()
plt.plot(epochs, val_accuracy, "bo", label="Validation accuracy")
plt.title("Validation Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, val_loss, "bo", label="Validation loss")
plt.title("Validation Loss")
plt.legend()
plt.show()
plt.plot(epochs, val_precision, "bo", label="Validation precision")
plt.title("Validation Precision")
plt.legend()
plt.show()

Model Inference

# Let's loads the best-performing model and evaluate on the test data

model = keras.models.load_model("decrease_image_checkpoint_filepath")
model.evaluate(test_generator)

50/50 [==============================] - 1s 16ms/step - loss: 0.1189 - accuracy: 0.9600 - precision_10: 0.9423

[0.11893752962350845, 0.9599999785423279, 0.942307710647583]

Decreasing the size of input images by almost 40% resulted in a training accuracy of 0.9650 and a validation accuracy of 0.99. The time it took to complete training and validation for 30 epochs was close to 3 minutes. The test performance is almost identical to the configuration with larger input sizes; the accuracy with the smaller image size is 0.96, while the accuracy with the larger image size is 0.9550.

Decreasing the image size and using the base configuration model outperformed all others in the experiments. Let's load the best model and continue exploring larger network architectures..

# Let's load the best performing model

best_model = keras.models.load_model("decrease_image_checkpoint_filepath")

Task 6 - Exploring off-the shelf tools

Teachable Machines

import matplotlib.image as mpimg

# Read the image
img = mpimg.imread("image.png")

# Create a new figure with a high DPI
plt.figure(dpi=300)  # Increase the value for higher quality images

# Display the image
plt.imshow(img)
plt.axis("off")  # Turn off the axis

# Show the plot with a high-quality image
plt.show()

Model Loading

# Let's set up the validation data generator
# Loads images of the same size and batch size as the training generator
valid_generator = my_generator.flow_from_directory(
    f"{base_directory}/validation/",
    target_size=(224, 224),
    batch_size=4,
    class_mode='binary'
)

# Let's set up the test data generator
# Uses the same parameters for consistency across training, validation, and testing
test_generator = my_generator.flow_from_directory(
    f"{base_directory}/test/",
    target_size=(224, 224),
    batch_size=4,
    class_mode='binary'
)

Found 200 images belonging to 2 classes.
Found 200 images belonging to 2 classes.

import numpy as np
from keras.models import load_model

# Load the model
model = load_model("keras_model.h5", compile=False)

# Load the labels
class_names = open("labels.txt", "r").readlines()

def predict_and_evaluate_from_generator(generator, num_batches):
    # To hold all predictions and true labels
    all_predictions = []
    all_true_labels = []

    # Loop over the generator to collect predictions
    for _ in range(num_batches):
        images, true_labels = next(generator)
        predictions = model.predict(images, verbose=0)
        predicted_labels = np.argmax(predictions, axis=1)
        all_predictions.extend(predicted_labels)
        all_true_labels.extend(true_labels)

    # Convert all predictions and true labels to numpy arrays
    all_predictions = np.array(all_predictions)
    all_true_labels = np.array(all_true_labels)

    # Calculate accuracy and precision
    accuracy = accuracy_score(all_true_labels, all_predictions)
    # Note: Set the average type according to your classification (binary, micro, macro, weighted, etc.)
    precision = precision_score(all_true_labels, all_predictions, average='binary')  # Adjust average as necessary

    # Print accuracy and precision
    print(f"Accuracy: {accuracy}")
    print(f"Precision: {precision}")

Validation Accuracy

num_batches = 5  # Define how many batches you want to use for evaluation
print("Evaluation on validation data:")
predict_and_evaluate_from_generator(valid_generator, num_batches)

Evaluation on validation data:
Accuracy: 0.7
Precision: 0.75

Test Accuracy

num_batches = 5  # Define how many batches you want to use for evaluation
print("Evaluation on test data:")
predict_and_evaluate_from_generator(test_generator, num_batches)

Evaluation on test data:
Accuracy: 0.7
Precision: 1.0

Using the Teachable Machines browser tool to fine-tune a political meme dataset resulted in a validation accuracy of 0.7. The test performance decreased compared to the base configuration; the accuracy with the Teachable Machines architecture was 0.7, while the accuracy with the base configuration was 0.9550.

Challenges in using Teachable Machines

1. Increasing the number of epochs frequently causes the web page to become unresponsive, necessitating repeated reloads of the website and re-uploads of images, which complicates the entire process.

2. After exporting the model and loading it using Keras to validate the validation and test accuracy, the accuracy changes every time I run the code. Typically, after training and loading a model, the validation and test accuracy should not vary.

ChatGPT

# Define the input shape and number of classes
input_shape = (150, 150, 3)
num_classes = 2  # for binary classification

# Start defining the model using the Functional API
inputs = keras.Input(shape=input_shape)
x = layers.Conv2D(32, 3, padding='same', activation='relu')(inputs)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(64, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(128, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(128, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Conv2D(256, 3, padding='same', activation='relu')(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Global Average Pooling serves as a way to reduce spatial dimensions to a single vector per feature map
x = layers.GlobalAveragePooling2D()(x)

# Output layer for binary classification
outputs = layers.Dense(1, activation='sigmoid')(x)

# Finalize and create the model
model = keras.Model(inputs=inputs, outputs=outputs)
model.summary()

Model: "model_7"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_8 (InputLayer)        [(None, 150, 150, 3)]     0         
                                                                 
 conv2d_47 (Conv2D)          (None, 150, 150, 32)      896       
                                                                 
 max_pooling2d_47 (MaxPooli  (None, 75, 75, 32)        0         
 ng2D)                                                           
                                                                 
 conv2d_48 (Conv2D)          (None, 75, 75, 64)        18496     
                                                                 
 max_pooling2d_48 (MaxPooli  (None, 37, 37, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_49 (Conv2D)          (None, 37, 37, 128)       73856     
                                                                 
 max_pooling2d_49 (MaxPooli  (None, 18, 18, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_50 (Conv2D)          (None, 18, 18, 128)       147584    
                                                                 
 max_pooling2d_50 (MaxPooli  (None, 9, 9, 128)         0         
 ng2D)                                                           
                                                                 
 conv2d_51 (Conv2D)          (None, 9, 9, 256)         295168    
                                                                 
 max_pooling2d_51 (MaxPooli  (None, 4, 4, 256)         0         
 ng2D)                                                           
                                                                 
 global_average_pooling2d_6  (None, 256)               0         
  (GlobalAveragePooling2D)                                       
                                                                 
 dense_15 (Dense)            (None, 1)                 257       
                                                                 
=================================================================
Total params: 536257 (2.05 MB)
Trainable params: 536257 (2.05 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

cb_early = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=30
)

# Let's compile the CNN model using binary cross_entropy as loss function and adam as optimizer
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy', keras.metrics.Precision()])

# Let's train and validate model on the training and validation data
history = model.fit(train_generator, validation_data = valid_generator, epochs = 30, verbose = 1, batch_size = 8, callbacks = [cb_early])

Epoch 1/30
150/150 [==============================] - 6s 27ms/step - loss: 0.6710 - accuracy: 0.6050 - precision_5: 0.6567 - val_loss: 0.4793 - val_accuracy: 0.7900 - val_precision_5: 0.9265
Epoch 2/30
150/150 [==============================] - 4s 25ms/step - loss: 0.6210 - accuracy: 0.7050 - precision_5: 0.6519 - val_loss: 0.4338 - val_accuracy: 0.9050 - val_precision_5: 0.8584
Epoch 3/30
150/150 [==============================] - 4s 26ms/step - loss: 0.4390 - accuracy: 0.8317 - precision_5: 0.7901 - val_loss: 0.5119 - val_accuracy: 0.7500 - val_precision_5: 0.6667
Epoch 4/30
150/150 [==============================] - 4s 25ms/step - loss: 0.3521 - accuracy: 0.8533 - precision_5: 0.8313 - val_loss: 0.2643 - val_accuracy: 0.8900 - val_precision_5: 0.8197
Epoch 5/30
150/150 [==============================] - 4s 25ms/step - loss: 0.3031 - accuracy: 0.8833 - precision_5: 0.8639 - val_loss: 0.3662 - val_accuracy: 0.8350 - val_precision_5: 0.7519
Epoch 6/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2957 - accuracy: 0.8933 - precision_5: 0.8806 - val_loss: 0.2907 - val_accuracy: 0.8550 - val_precision_5: 0.7752
Epoch 7/30
150/150 [==============================] - 4s 26ms/step - loss: 0.2468 - accuracy: 0.9017 - precision_5: 0.8951 - val_loss: 0.2584 - val_accuracy: 0.9150 - val_precision_5: 0.9029
Epoch 8/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2746 - accuracy: 0.8950 - precision_5: 0.8990 - val_loss: 0.3227 - val_accuracy: 0.8500 - val_precision_5: 0.7734
Epoch 9/30
150/150 [==============================] - 4s 26ms/step - loss: 0.2190 - accuracy: 0.9233 - precision_5: 0.9123 - val_loss: 0.2483 - val_accuracy: 0.8950 - val_precision_5: 0.8264
Epoch 10/30
150/150 [==============================] - 4s 26ms/step - loss: 0.2082 - accuracy: 0.9200 - precision_5: 0.9257 - val_loss: 0.4277 - val_accuracy: 0.8100 - val_precision_5: 0.7246
Epoch 11/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1835 - accuracy: 0.9317 - precision_5: 0.9331 - val_loss: 0.2966 - val_accuracy: 0.8850 - val_precision_5: 0.8130
Epoch 12/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1805 - accuracy: 0.9233 - precision_5: 0.9291 - val_loss: 0.3917 - val_accuracy: 0.8450 - val_precision_5: 0.7634
Epoch 13/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1730 - accuracy: 0.9400 - precision_5: 0.9400 - val_loss: 0.1531 - val_accuracy: 0.9400 - val_precision_5: 0.9000
Epoch 14/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1709 - accuracy: 0.9333 - precision_5: 0.9305 - val_loss: 0.3064 - val_accuracy: 0.8500 - val_precision_5: 0.7692
Epoch 15/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1528 - accuracy: 0.9400 - precision_5: 0.9400 - val_loss: 0.1737 - val_accuracy: 0.9300 - val_precision_5: 0.8772
Epoch 16/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1652 - accuracy: 0.9300 - precision_5: 0.9358 - val_loss: 0.2502 - val_accuracy: 0.8950 - val_precision_5: 0.8264
Epoch 17/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1328 - accuracy: 0.9483 - precision_5: 0.9498 - val_loss: 0.3321 - val_accuracy: 0.8750 - val_precision_5: 0.8000
Epoch 18/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0989 - accuracy: 0.9617 - precision_5: 0.9632 - val_loss: 0.3960 - val_accuracy: 0.8900 - val_precision_5: 0.8197
Epoch 19/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0944 - accuracy: 0.9617 - precision_5: 0.9695 - val_loss: 0.3759 - val_accuracy: 0.8700 - val_precision_5: 0.7937
Epoch 20/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0900 - accuracy: 0.9650 - precision_5: 0.9697 - val_loss: 0.2035 - val_accuracy: 0.9200 - val_precision_5: 0.8818
Epoch 21/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0877 - accuracy: 0.9667 - precision_5: 0.9730 - val_loss: 0.3680 - val_accuracy: 0.8700 - val_precision_5: 0.7937
Epoch 22/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0883 - accuracy: 0.9717 - precision_5: 0.9732 - val_loss: 0.4260 - val_accuracy: 0.8550 - val_precision_5: 0.7752
Epoch 23/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0559 - accuracy: 0.9833 - precision_5: 0.9801 - val_loss: 0.4858 - val_accuracy: 0.9050 - val_precision_5: 0.8403
Epoch 24/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0802 - accuracy: 0.9633 - precision_5: 0.9572 - val_loss: 0.1835 - val_accuracy: 0.9400 - val_precision_5: 0.8929
Epoch 25/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0745 - accuracy: 0.9733 - precision_5: 0.9702 - val_loss: 0.4261 - val_accuracy: 0.8700 - val_precision_5: 0.7937
Epoch 26/30
150/150 [==============================] - 4s 25ms/step - loss: 0.0477 - accuracy: 0.9867 - precision_5: 0.9834 - val_loss: 0.6151 - val_accuracy: 0.8500 - val_precision_5: 0.7692
Epoch 27/30
150/150 [==============================] - 4s 26ms/step - loss: 0.0682 - accuracy: 0.9817 - precision_5: 0.9801 - val_loss: 0.2826 - val_accuracy: 0.9000 - val_precision_5: 0.8333
Epoch 28/30
150/150 [==============================] - 4s 24ms/step - loss: 0.2215 - accuracy: 0.9417 - precision_5: 0.9344 - val_loss: 0.1733 - val_accuracy: 0.9150 - val_precision_5: 0.8547
Epoch 29/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1016 - accuracy: 0.9633 - precision_5: 0.9664 - val_loss: 0.4181 - val_accuracy: 0.8450 - val_precision_5: 0.7634
Epoch 30/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1114 - accuracy: 0.9567 - precision_5: 0.9507 - val_loss: 0.1700 - val_accuracy: 0.9450 - val_precision_5: 0.9009

Model Evaluation(Precison, Recall, F1-score) - ChatGPT

train_accuracy = history.history["accuracy"]
train_loss = history.history["loss"]
train_precision = history.history["precision_5"]
val_accuracy = history.history["val_accuracy"]
val_loss = history.history["val_loss"]
val_precision = history.history["val_precision_5"]
epochs = range(1, len(train_accuracy) + 1)
plt.plot(epochs, train_accuracy, "bo", label="Training accuracy")
plt.title("Training Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, train_loss, "bo", label="Training loss")
plt.title("Training Loss")
plt.legend()
plt.show()
plt.plot(epochs, train_precision, "bo", label="Training precision")
plt.title("Training Precision")
plt.legend()
plt.show()
plt.plot(epochs, val_accuracy, "bo", label="Validation accuracy")
plt.title("Validation Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, val_loss, "bo", label="Validation loss")
plt.title("Validation Loss")
plt.legend()
plt.show()
plt.plot(epochs, val_precision, "bo", label="Validation precision")
plt.title("Validation Precision")
plt.legend()
plt.show()

Model Inference

# Let's test the model
model.evaluate(test_generator)

50/50 [==============================] - 1s 17ms/step - loss: 0.3606 - accuracy: 0.9050 - precision_5: 0.8403

[0.36062026023864746, 0.9049999713897705, 0.8403361439704895]

Using the code provided by ChatGPT to build, compile, and fine-tune a political meme dataset resulted in a validation accuracy of 0.9450. The test performance was almost similar to the base configuration; the accuracy with the ChatGPT architecture was 0.91, while the accuracy with the base configuration was 0.9550.

Challenges in using ChatGPT

1. Using ChatGPT to train and validate a political images dataset resulted in code that had several issues, particularly with the incorrect number of neurons in the dense layer, leading to repeated graph execution errors. Despite multiple attempts to resolve the issue through ChatGPT, the problem persisted, and I ultimately had to fix it myself.

2. After resolving all the issues for training, the validation accuracy fluctuated randomly. When asked, ChatGPT produced different versions of code that did not make any sense. Furthermore, the explanations it provided for the fluctuations were not logical.

Gemini

from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense


# Define the model architecture
model_gemini = Sequential()
model_gemini.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(150, 150, 3)))  # Update input shape
model_gemini.add(MaxPooling2D((2, 2)))
model_gemini.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))
model_gemini.add(MaxPooling2D((2, 2)))
# model.add(Flatten())
model_gemini.add(layers.GlobalAveragePooling2D())
model_gemini.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))
model_gemini.add(Dense(1, activation='sigmoid'))  # Output layer for binary classification

# Compile the model
model_gemini.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy', keras.metrics.Precision()])

# Train the model
history_gemini = model_gemini.fit(train_generator, epochs=30, validation_data=valid_generator)

Epoch 1/30
150/150 [==============================] - 5s 27ms/step - loss: 0.6630 - accuracy: 0.5733 - precision_8: 0.6134 - val_loss: 0.5693 - val_accuracy: 0.7100 - val_precision_8: 0.6750
Epoch 2/30
150/150 [==============================] - 4s 26ms/step - loss: 0.5577 - accuracy: 0.7150 - precision_8: 0.7416 - val_loss: 0.4817 - val_accuracy: 0.8150 - val_precision_8: 0.7890
Epoch 3/30
150/150 [==============================] - 4s 26ms/step - loss: 0.4771 - accuracy: 0.7900 - precision_8: 0.7939 - val_loss: 0.3996 - val_accuracy: 0.8400 - val_precision_8: 0.7881
Epoch 4/30
150/150 [==============================] - 4s 25ms/step - loss: 0.3992 - accuracy: 0.8250 - precision_8: 0.8095 - val_loss: 0.3371 - val_accuracy: 0.8700 - val_precision_8: 0.7984
Epoch 5/30
150/150 [==============================] - 4s 25ms/step - loss: 0.3558 - accuracy: 0.8483 - precision_8: 0.8276 - val_loss: 0.4387 - val_accuracy: 0.7750 - val_precision_8: 0.6897
Epoch 6/30
150/150 [==============================] - 4s 25ms/step - loss: 0.3103 - accuracy: 0.8817 - precision_8: 0.8658 - val_loss: 0.4433 - val_accuracy: 0.7700 - val_precision_8: 0.6849
Epoch 7/30
150/150 [==============================] - 4s 25ms/step - loss: 0.3263 - accuracy: 0.8667 - precision_8: 0.8526 - val_loss: 0.5119 - val_accuracy: 0.7300 - val_precision_8: 0.6494
Epoch 8/30
150/150 [==============================] - 4s 26ms/step - loss: 0.2756 - accuracy: 0.9050 - precision_8: 0.8932 - val_loss: 0.2183 - val_accuracy: 0.9200 - val_precision_8: 0.8684
Epoch 9/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2642 - accuracy: 0.8967 - precision_8: 0.8839 - val_loss: 0.3593 - val_accuracy: 0.8150 - val_precision_8: 0.7299
Epoch 10/30
150/150 [==============================] - 4s 26ms/step - loss: 0.2611 - accuracy: 0.9050 - precision_8: 0.8958 - val_loss: 0.4499 - val_accuracy: 0.7800 - val_precision_8: 0.6944
Epoch 11/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2403 - accuracy: 0.9117 - precision_8: 0.9049 - val_loss: 0.2149 - val_accuracy: 0.9200 - val_precision_8: 0.8621
Epoch 12/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2361 - accuracy: 0.9100 - precision_8: 0.9100 - val_loss: 0.3915 - val_accuracy: 0.8300 - val_precision_8: 0.7463
Epoch 13/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2278 - accuracy: 0.9100 - precision_8: 0.9046 - val_loss: 0.2466 - val_accuracy: 0.9100 - val_precision_8: 0.8475
Epoch 14/30
150/150 [==============================] - 4s 26ms/step - loss: 0.2473 - accuracy: 0.9117 - precision_8: 0.9158 - val_loss: 0.4923 - val_accuracy: 0.7800 - val_precision_8: 0.6944
Epoch 15/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2326 - accuracy: 0.9050 - precision_8: 0.9037 - val_loss: 0.2016 - val_accuracy: 0.9250 - val_precision_8: 0.8696
Epoch 16/30
150/150 [==============================] - 4s 26ms/step - loss: 0.2178 - accuracy: 0.9217 - precision_8: 0.9148 - val_loss: 0.2152 - val_accuracy: 0.9200 - val_precision_8: 0.8621
Epoch 17/30
150/150 [==============================] - 4s 27ms/step - loss: 0.2037 - accuracy: 0.9183 - precision_8: 0.9226 - val_loss: 0.2887 - val_accuracy: 0.8800 - val_precision_8: 0.8065
Epoch 18/30
150/150 [==============================] - 4s 27ms/step - loss: 0.2042 - accuracy: 0.9117 - precision_8: 0.9076 - val_loss: 0.2837 - val_accuracy: 0.8600 - val_precision_8: 0.7812
Epoch 19/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1958 - accuracy: 0.9350 - precision_8: 0.9394 - val_loss: 0.3034 - val_accuracy: 0.8600 - val_precision_8: 0.7812
Epoch 20/30
150/150 [==============================] - 4s 27ms/step - loss: 0.1888 - accuracy: 0.9267 - precision_8: 0.9267 - val_loss: 0.2670 - val_accuracy: 0.8900 - val_precision_8: 0.8197
Epoch 21/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1981 - accuracy: 0.9267 - precision_8: 0.9324 - val_loss: 0.1672 - val_accuracy: 0.9300 - val_precision_8: 0.8772
Epoch 22/30
150/150 [==============================] - 4s 25ms/step - loss: 0.2104 - accuracy: 0.9100 - precision_8: 0.9184 - val_loss: 0.2236 - val_accuracy: 0.9150 - val_precision_8: 0.8547
Epoch 23/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1802 - accuracy: 0.9317 - precision_8: 0.9390 - val_loss: 0.4033 - val_accuracy: 0.8450 - val_precision_8: 0.7634
Epoch 24/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1950 - accuracy: 0.9267 - precision_8: 0.9267 - val_loss: 0.2226 - val_accuracy: 0.9150 - val_precision_8: 0.8547
Epoch 25/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1764 - accuracy: 0.9367 - precision_8: 0.9549 - val_loss: 0.2855 - val_accuracy: 0.8850 - val_precision_8: 0.8130
Epoch 26/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1805 - accuracy: 0.9367 - precision_8: 0.9486 - val_loss: 0.1975 - val_accuracy: 0.9250 - val_precision_8: 0.8696
Epoch 27/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1731 - accuracy: 0.9367 - precision_8: 0.9456 - val_loss: 0.2244 - val_accuracy: 0.9100 - val_precision_8: 0.8475
Epoch 28/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1756 - accuracy: 0.9333 - precision_8: 0.9452 - val_loss: 0.4865 - val_accuracy: 0.8150 - val_precision_8: 0.7299
Epoch 29/30
150/150 [==============================] - 4s 26ms/step - loss: 0.1713 - accuracy: 0.9317 - precision_8: 0.9390 - val_loss: 0.4026 - val_accuracy: 0.8450 - val_precision_8: 0.7634
Epoch 30/30
150/150 [==============================] - 4s 25ms/step - loss: 0.1789 - accuracy: 0.9350 - precision_8: 0.9516 - val_loss: 0.2161 - val_accuracy: 0.9300 - val_precision_8: 0.8772

Model Evaluation(Precison, Recall, F1-score) - Gemini

train_accuracy = history_gemini.history["accuracy"]
train_loss = history_gemini.history["loss"]
train_precision = history_gemini.history["precision_8"]
val_accuracy = history_gemini.history["val_accuracy"]
val_loss = history_gemini.history["val_loss"]
val_precision = history_gemini.history["val_precision_8"]
epochs = range(1, len(train_accuracy) + 1)
plt.plot(epochs, train_accuracy, "bo", label="Training accuracy")
plt.title("Training Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, train_loss, "bo", label="Training loss")
plt.title("Training Loss")
plt.legend()
plt.show()
plt.plot(epochs, train_precision, "bo", label="Training precision")
plt.title("Training Precision")
plt.legend()
plt.show()
plt.plot(epochs, val_accuracy, "bo", label="Validation accuracy")
plt.title("Validation Accuracy")
plt.legend()
plt.figure()
plt.plot(epochs, val_loss, "bo", label="Validation loss")
plt.title("Validation Loss")
plt.legend()
plt.show()
plt.plot(epochs, val_precision, "bo", label="Validation precision")
plt.title("Validation Precision")
plt.legend()
plt.show()

Model Inference

# Let's test the model
model_gemini.evaluate(test_generator)

50/50 [==============================] - 1s 16ms/step - loss: 0.3590 - accuracy: 0.8750 - precision_8: 0.8000

[0.3590352535247803, 0.875, 0.800000011920929]

Using the code provided by Gemini to build, compile, and fine-tune a political meme dataset resulted in a validation accuracy of 0.93. The test performance was almost similar to the base configuration; the accuracy with the Gemini architecture was 0.88, while the accuracy with the base configuration was 0.9550.

Challenges in using Gemini

1. Using Gemini to train and validate a political images dataset resulted in code that had several issues, particularly with the incorrect number of neurons in the dense layer, leading to repeated graph execution errors. Despite multiple attempts to resolve the issue through Gemini, the problem persisted, and I ultimately had to fix it myself.

2. After resolving all the issues for training, the validation accuracy fluctuated randomly. When asked, Gemini produced different versions of code that did not make any sense. Furthermore, the explanations it provided for the fluctuations were not logical.

Comparison of Different Off Shelf Tools on Model Performance

Tool	Validation Accuracy	Test Accuracy
Teachable Machines	0.7	0.7
ChatGPT	0.9450	0.91
Gemini	0.93	0.88